arXiv:1503.02850vl [cond-mat.stat-mech] 10 Mar 2015 


Manuscript prepared for Annales Geophysicae 
with version 3.2 of the class copernicus.cls. 
Date: 11 March 2015 


Kinetic theory of information - the dynamics of information 

R. A. Treumann**’^ and W. Baumjohaim^ 

'Department of Geophysics and Environmental Sciences, Munich University, Munich, Germany 

^International Space Science Institute, Bern, Switzerland 

■'Space Research Institute, Austrian Academy of Sciences, Graz, Austria 

* Visiting the International Space Science Institute, Bern, Switzerland 


Abstract 

A kinetic approach to the notion of information is proposed, 
based on Liouville kinetic theory. The general kinetic equa¬ 
tion for the evolution of the N-particle information Im in a 
Hamiltonian system of large particle number ^ 1 is ob¬ 
tained. It is shown that the A^-particle information is strictly 
conserved. Defining reduced particle number information 
densities in phase space should be possible to obtain a kinetic 
equation for the ordinary one-particle information Xi=X 
following the Bogoliubov prescription. The kinetic equation 
for I is a kind of generalized Boltzmann equation with in¬ 
teraction term depending on the hierarchy of reduced infor¬ 
mations. This term in its general form is the most general 
expression for the Kolmogorov entropy rate of evolution of 
the information. 

Keywords. Statistical mechanics. Entropy, Liouville theory. 
Data analysis 


Introduction 

The internal information content of a physical system is its 
entropy S which, in classical Hamiltonian systems with one- 
particle (index 1) Hamiltonian = ^^(p^/2m)-f 

^ function of the particle coordinate vec¬ 
tors in Boltzmann’s 6-dimensional (3 time dependent 
momentum vector pit) and 3 space vector q{t) coordinates, 
with t time) one-particle /i-phase space, mass m and poten¬ 
tial energy Uiqj), respectively, with i,j particle number. It 
depends on the complete dynamics of all indistinguishable 
particles on the kinetic level under the action of the inter¬ 
particle forces (accounted for in the potential U) which con¬ 
tribute to iJi and are defined by the Hamilton equations of 
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motion q = dHi/dp,p = —dHi/dq. This internal informa¬ 
tion is, according to Boltzmann and Shannon, given as the 
product of the p-phase space density Fi{p,q,t), the one- 
particle distribution function, and its logarithm logFi, with 
Fi satisfying the Boltzmann equation dtFi -f [Hi,Fi] = Cb, 
with = {dpF[i){dqFi) — idqF{i)idpFi) the one parti¬ 

cle Poisson bracket, and Cb Boltzmann’s collision integral. 

In any realistic classical physical system composed of 
N subsystems (particles) of very large number 3> 1, the 
Boltzmann equation is replaced by the Liouville equation 

CnF]s[ = 0, with Cn =+ (1) 


which holds in the 6A^-dimensi onal Gibbs’ E- phase space 
(for a general reference cf., e.g., Balescu . 2000[ including a 
discussion of quantum systems). The operator Cn is the N- 
particle Liouville operator. It contains the A^-particle Hamil¬ 
tonian Hn{p^, qN), which in classical theory is not an op¬ 
erator. Here the brackets are A^-particle Poisson’s 

brackets. 

This Liouville equation, unlike the Boltzmann equation, 
is exact. It acts on the exact phase space density, which for 
classical point particles is defined as 


N 

L’Ar((7,p,f) =]^(5[g-g,(f)](5[p-p,(f)] 


I 


( 2 ) 


All dynamics is contained in the time-dependent phase space 
coordinates of the N particles (subsystems) qi{t),piit) via 
Hamilton’s equations. F^, being a phase space (probabil¬ 
ity) density, must be normalised accordingly. In this sense, 
Liouville’s and Hamilton’s equations are tautologies. How¬ 
ever, the former offers the advantage of a probabilistic ap¬ 
proach which avoids the necessity of solving for all N 
fully dynamical Hamilton equations qN = dH]sr/dpN,PN = 
-dHM/dqN- 

The Liouville equation describes the flow of the N parti¬ 
cles through A^-particle phase-space under the action of the 
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A^-particle Hamiltonian, explicitly exhibiting the conserva¬ 
tion of particles. Its (formally known) solution though 
exact is rather impractical. It requires knowledge of all N ex¬ 
act particle orbits at all times t. The Liouville equation can, 
however, be reduced to a one-particle Boltzmann-like equa¬ 
tion holding in one-particle phase space (with all particles 
identical distinguished only by mass, charge, and energy) 
following a complicated procedure of reductive integration 
known as the BBGKY hierarchy (after N. N. Bogoliubov, M. 
Born, H. S. Gr een, J. G. Kirkwood , and J. Yvonn e) (for ref¬ 
erence cf., e.g. Bogolvubovl 19621: Huangl 1987h . We will 
refer to an equivalent of this approach below in Section 3. 

The question which interests us here concerns the nature 
of entropy/information on the N -particle level in A^-particle 
Gibbs’ T-phase space. Answering this question should pro¬ 
vide an evolution theory for the information of a physical sys¬ 
tem. In other words, one may hope obtaining a Boltzmann- 
like one-particle p-phase space kinetic equation for the infor¬ 
mation. 

This question is not purely academic. Interest in the 
physics of information arose primarily with the advent of 
chaos theory in the early sixties (for a collection of differ¬ 
ent approaches in different fiel ds the r eader is referred to the 
Santa Fe proceedings volume zurek , 1990l) . Indeed, infor¬ 
mation is one of the central quantities not only in physics 
but also in several other fields like communication, biology 
etc. all referring in one or the other sense to information 
theory. Information theory makes use of a physical defini¬ 
tion of information that is exploited for instance in maximum 
entropy methods of data analysis as also in various applica¬ 
tions to the determination of probability measures, chaotic 
behaviour, as well as quite practical problems like weather 
prediction in meteorology, climate research, prediction the¬ 
ory in the evolution of time series, in space physics in general 
and space weather in particula r (in the latter r espec t see, e.g., 
the comprehensive review in iBalasis et al.l 1201 3l and ref¬ 
erences therein) where low-dimensional chaotic approaches 
have found wide application. Thermodynamics predicts that 
information cannot be lost. Under stationary conditions it en¬ 
ters thermodynamics respectively statistical mechanics being 
the central quantity in the first and second thermodynamic 
laws where it relates directly to internal energy and external 
work done on the system. Under non-stationary conditions 
its evolution is barely known while being of utmost impor¬ 
tance. Since thermodynamics is rooted in kinetic theory it is 
reasonable to ask whether information cannot be subject to 
kinetic theory as well. 

So far, chaos theory of low-dimensional systems provides 
tools already to infer about generation of information and its 
effects on the system. However, most physical systems are 
composed of very many subsystems the interaction of which 
should be taken into account when considering information, 
inferring about its evolution, production and accumulation. 
Such a theory should be rooted in first physical principles, 
i.e. for systems consisting of many subsystems and having 


large numbers of degrees of freedom it should be based on 
Liouville theory in order to make information accessible to 
well known physical methods, technical treatment and pro¬ 
viding a deeper understanding of its evolution and distribu¬ 
tion throughout the system and sharing by the various sub¬ 
systems. Since information, once generated, cannot be lost, 
such distribution processes are of importance in particular 
in view of processes which, like information spread in soci¬ 
eties, so far are not subject to any treatment in physics. In 
such systems apparently very small amounts of energy are 
involved causing large effects by information transfer. In the 
following we briefly sketch how a time-dependent physical 
information theory could be developed. 


N-particle Information 

Let us define an equivalent Ai-particle phase-space informa¬ 
tion density following the Boltzmann-Shannon prescrip¬ 
tion 


1^n{Fn) = ^tvlogFjv 


(3) 


assuming that it is also normalized (for instance to Boltz¬ 
mann’s constant ks)- Via the phase space density Fm it de¬ 
pends on the complete phase space dynamics of the system 
contained in the Hamiltonian and Hamilton’s equations. The 
question is then, which equation does the exact A^-particle 
information satisfy? 

In order to answer this question we tentatively apply the 
above Liouville operator Cn to Im keeping in mind that it 
applies strictly only to the At-particle phase space distribu¬ 
tion function Fjy. Before doing this we rewrite Xm using the 
definition of the exact phase space density Equation (|2]) 

XN{FN) = Y[6{q-qi)S{p-p,)logY[S{q-qj)S{p-pj)(4) 

i 3 

the logarithm of the product becomes a sum X]j^g(5(q — 
qj)S{p — pj). Mathematically the logarithm of a distribu¬ 
tion - the delta function - must be taken with care; this is a 
weakness in the Shannon definition on the microscopic level. 
Hence the expression is to be considered as a formal repre¬ 
sentation only. Formally, however, the delta functions take 
care for that all mixed products vanish. Hence one finds the 
obvious result that 

N 

lN = '^Xi with Ii = Fi\ogFi (5) 

i 


is the sum of all single particle informations. In other words, 
the A^-particle information is additive (extensive). This is a 
consequence of the logarithmic dependence imposed by the 
Shannon prescription to which we restrict here. Other defini¬ 
tions as, for instan ce, the g eneralisations available in the liter¬ 


ature (for instance iRenv: 


f 


or that given recently in 


1 97(11: IWehrlL ll 9781: iTsalli^ [l 988|i 
Treumann & BaumjohannI ( 2014 ) 
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may destroy the extensivity already on the Liouville level by 
adding correlations in the last expression. 

If we apply the Liouville operator to Equations (|3]l and 
it is easy to demonstrate for any conservative system that 

= (1 + logXAr)£Ari^Ar = 0 (6) 

which holds for any physically reasonable logXAr 7^—1 and 
because of Equation ([T]). Eor this reason the exact A^-particle 
information satisfies the exact iV-particle Liouville equation 

Mtv(X^) = 0 (7) 

The 7V-particle information thus follows Liouville dynamics 
in phase space with the dynamic equations prescribed by the 
7V-particle Hamiltonian function. We may note here that this 
is a classical and no quantum theory. Generalisation to quan¬ 
tum theory is by no means obvious. The nonlinear nature of 
the information inhibits the simple replacement of Poisson’s 
brackets with commutators/anti-commutators. 

Within Shannon theory we therefore find an exact kinetic 
equation for the exact classical 7V-particle Shannon informa¬ 
tion In, and that this equation is the Liouville equation (|7]) 
acting on the full A^-particle phase space. In principle, this is 
an expected result. 

However, like in the case of exact kinetic particle theory, 
this equation is an identity and not yet an equation with that 
one could operate. It just says that in the exact W-particle 
phase space the iV-particle information is conserved and be¬ 
haves like a phase space information density which corre¬ 
sponds to an information flow through phase space. In fact 
this is not such a surprise. Since no averaging has been done 
when following the exact orbits of all particles, no overall 
disorder of the phase space is produced. The total informa¬ 
tion content which has been there in the total phase space 
volume at the beginning is still there; it is simply conserved. 
Information could not have gone anywhere from the total 
volume. 

It is again noteworthy that this conclusion is strictly valid 
only for the A^-particle Shannon information. Other defini¬ 
tions might not reproduce Liouville’s equation in its known 
form with vanishing right hand side. In the general case one 
expects that an iV-particle diffusion term occurs on the right 
in this case being responsible for the dispersion of the gen¬ 
eralised information in A^-particle phase space and violat¬ 
ing the extensivity of the iV-particle information expressed 
in Equ. (|7]i. If this is the case, there must be some physical 
reason imposed from the outside for the definition of infor¬ 
mation at variance. Non-conservation of information then 
means that loss or gain of total information is attributed to 
this reason. One may, for instance, think of interaction with 
some external field which, on the global level, extracts infor¬ 
mation from the system or adds information to it. We will 
not consider this case here. 


Hierarchy 


Physical reality as experienced in practice does, however, 
not take place on the microscopic level of A^-particle phase 
space. The path to a practical kinetic theory of (Shannon) 
information is in principle prescribed by the analogy to ki¬ 
netic theory. There a hierarchy of average n= {N — j)- 
particle distributions Fn, with j G N,j < TV — 1 is prescribed 
as the suitably normalised integral over the phase space 
coordinates of all particles j. This procedure, known as 
BBGKY hierarchy, ultimately reduces the phase space to the 
one-particle /i-phase space of the now undistinguishable TV 
particles. It in this case produces the average one-particle 
phase space distribution function f{q,p) = Fi and, from Li¬ 
ouville’s equation, the ultimate one-particle kinetic equa¬ 
tion describing the evolution of f{q,p) under the action of 
the reduced one-particle Hamiltonian Hi{q,p). This final 
one-particle Boltzmann-like kinetic equation contains a non¬ 
vanishing right-hand side which collects all correlations be- 
twe en the particles and their mutual interaction f i elds (cf. , 
e.g, Montgomery & Tidmanl 1964 Reichll 198(i Balescu . 
200(1 for their explicit classical and quantum forms). In case 


of merely hard-core binary collisions, this term reduces fur¬ 
ther to Boltzmann’s /i-space equation which, in the total ab¬ 
sence of any collisions and just for purely classical field in- 
teractions, becomes th e zero-right hand side Vlasov equation 
( Klimontovichi 19671 Ch. 3) or, including quasilinear inter¬ 
actions with self-excited field fluctuations in the one-particle 
kinetic equation, the Eokker-Planck equation (for a rigorous 
and lucid derivation of the Eok ker-Planck equations cf., e.g., 
Montgomery & Tidmanl 1964 ). 


A similar procedure should go along the lines of an analo¬ 
gous definition of phase-space-averaged “reduced" informa¬ 
tions In forming a descending in n chain. The philosophy 
behind this approach is that the Shannon information is un¬ 
derstood as the average of the logarithm of the distribution 
function itself. Thus any reduced information is given as 

/d3(”+i)gd3("+i)pL’„+i(g,p,f)logF„+i(g,p,f) 

/d3("+i)9d3("+i)pX„+i 


The problem consists in finding the kinetic equation that gov¬ 
erns the evolution of X„ with n^l from the exact Liouville 
equation for In- This step is substantially more complex 
and less transparent than in the case of the BBGKY hierar¬ 
chy of the n-particle distribution function which results in the 
Boltzmann equation. There the nonlinearity is provided by 
the Hamiltonian, while in the case of the information the n- 
particle information itself is intrinsically nonlinear, and care 
must be taken in each step when applying the reduced Hamil¬ 
tonians. 

Carrying through this program is a formidable task though 
being quite straightforward. In every step of the reduction 
procedure one has to take care of the reduced Liouville oper¬ 
ator, the hierarchy equation of the former step for the reduced 
distribution function. Already in the BBGKY hierarchy a 
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non-vanishing and rather complicated term on the right-hand 
side of the reduced kinetic equation is produced to which 
each reduction step adds further terms. This will also hap¬ 
pen in the case of the reduction of the Liouville equation (I7]i 
for the information. In this note we refrain from performing 
all these steps leaving it to someone else who can explicit it. 
The present paper is just a perspective paper that intends to 
present the basic idea of constructing a viable physical the¬ 
ory of the dynamics of information. However, some simple 
arguments can be given what the reduction procedure, the 
information hierarchy, will lead us to. 

In the case of the BBGKY hierarchy the form of the Liou¬ 
ville equation on the left hand side of the equation reproduces 
in each step with descending subsystem (particle) number n 
from n = N down to n = 1, the wanted hnal form of the 
kinetic equation. All correlations and subsystem (particle) 
interactions become relegated to form an ever more compli¬ 
cated interaction term on the right hand side of these equa¬ 
tions. Because the structure of the A^-subsystem Liouville 
equation for the A^-subsystem information In is identical 
to that of the A^-particle Liouville equation for Fn, we ex¬ 
pect that this behaviour will reproduce also in the case of 
the information hierarchy. We conjecture that the hnal one- 
subsystem (particle) kinetic equation for the information will 
be of the form 


that this is particular only to the Boltzmann-Shannon infor¬ 
mation. Other dehnitions of entropy-information found in 
the literature might not lead to similar kind of reductions. 
Having conjectured the form of the one-subsystem kinetic 
equation for the one-subsystem information and determined 
the implicit functional form of the correlation term on the 
right of the above equation, will provide a full kinetic theory 
of the evolution of information in an 7V-subsystem conhgura- 
tion. This is most interesting for a number of obvious reasons 
and may apply not only to physics but also to other sciences 
and engineering where information generation and evolution 
plays an important role. Here we have restricted ourselves to 
physics alone. 

To close this perspective article we point out that the above 
conjectured kinetic equation for the one-subsystem informa¬ 
tion can also be understood differently. The reduced (one- 
subsystem) Liouville operator on the left can be written as 
the total time derivative in the one-subsystem phase space. 

The kinetic equation then reads 

Cili{pi,qi) = {t)=C{qi{t),...qN{t)]Pi{t),...pN[t)}{\Q) 

The formal solution of this version is 

Ii{t)= f dtC{qi{t),...qNit);piit),...pNit)} (11) 

J —OO 


CiIi{pi,qi)=C{qi,q 2 ...qN',Pl,P 2 ---PN} 


(9) 


where Ci= C = dt + [Hi{p,q),...], with p = pi,q = qi and 
Hi {p,q) the one-particle Hamiltonian, is the one-particle Li¬ 
ouville operator, and C, as indicated by the braces {...}, 
functionally contains all coordinates and correlations. It 
results in the reductive procedure leading from the N- to 
the one-subsystem kinetic equation. All the dynamics that 
causes the evolution of the information in the interaction 
between the particles is contained in C. Since the Hamil¬ 
tonian also contains collisionless interactions with external 
and self-consistent fields, such interactions are taken care on 
the left-hand side of the above kinetic equation. It is, how¬ 
ever, questionable whether neglect of the right-hand side can 
be justihed in the case of information as is done in Vlasov 
and Klimontovich (IKlimontovichi 1 9671) theory. Presumably, 
since information (entropy) cannot be erased but can only 
grow, no comparably simple argument can be found for drop¬ 
ping C in the case of information unless the system is in ther¬ 
modynamic equilibrium when its dynamics and evolution is 
obsolete. 


Discussion 


which yields the time evolution of the one-subsystem in¬ 
formation. The difficulty here lies not only in the neces¬ 
sity to know the explicit functional form of C which fol¬ 
lows from the hierarchy approach, it is relegated in addi¬ 
tion to the knowledge of time dependence of the one-, two-, 
... subsystem trajectories in phase space, which strictly spo¬ 
ken implies the knowledge of the full particle dynamics in 
phase space which are not known a priori. Thus the above 
formal solution is a tautology, and one has to apply some 
kind of approximation like perturbation methods to treat the 
perturbation of an initial state. This resembles the situation 
encountered in kinetic theory. Nevertheless, though this is 
not completely satisfactory, the above form shows that the 
total time derivative of the one-subsystem information is de¬ 
termined by the functional C. Conventionally, in the the¬ 
ory of chaotic interactions of a small number of particles 
this is taken to be the so-cal led Kolmogorov entropy rate 
K ( Kolmogorov , 19581 1959h which originally appeared as 
a metric entropy rate in Kolmogorov-Arnold-Moser ( KAM 
theory of chaotic processes (for reference cf., e.g. iBroei 
20041) . Hence, performing the derivation of the hierarchy 
leads in a straight way to the physical definition of the Kol¬ 
mogorov K entropy rate for an 7V-subsystem conhguration 
as 




Presumably, the dynamical theory of Boltzmann-Shannon in¬ 
formation will result in a Boltzmann-like kinetic equation for 
the one-subsystem information Xi with, however, very com¬ 
plicated non-vanishing correlation term on its right. It is clear 


K{t)=C{qi{t),...qN{t);pi{t),...pN{t)} (12) 

as an implicit function of time and the number of subsys¬ 
tems (particles). For any non-stationary system C^O for the 
simple reason that entropy/information can only grow. Since 
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C itself depends implicitly on the higher order informations, 
the Kolmogorov entropy rate is also a functional of informa¬ 
tion. Moreover, t=I~^ is the (properly normalised) inverse 
functional of the information. One thus writes 

dI/dt = K{I} (13) 

where, for simplicity, we dropped the index 1. From here we 
obtain the interesting expression 

dt = dI/K{I} (14) 

with its formal solution 

t-to= [ dI/K{I} (15) 

Jlo 

yielding the time elapsed during the evolution of the infor¬ 
mation from state Xq to X. This solution suggests that for 
K = C = 0 the elapsed time is infinite, which is nothing 
else but another expression for that the information does not 
change but is conserved. Otherwise, for K ^ oo the sys¬ 
tem seems to evolve at a diverging rate. However, inspection 
of the second last equation reveals that df = 0 in this case, 
and the entire expression becomes obsolete. This case corre¬ 
sponds to complete stochasticity with no production nor evo¬ 
lution of information at all, the final thermodynamic equilib¬ 
rium state of maximum information. 

The important case is the intermediate one where K is 
finite, corresponding to a state of nonlinear non-stochastic 
(chaotic) interactions. They cause the information to evolve 
in finite time. Speculatively, this expression can be taken 
as an equation for the “production of time". In such an in¬ 
terpretation, time is “generated" under circumstances when 
information is produced - a physically not unreasonable as¬ 
sertion. In this interpretation there is more room left for spec¬ 
ulation. The case K = 0 corresponds to stationarity with no 
time evolution of information. Information can only be redis¬ 
tributed convectively then. K = oo means complete stochas¬ 
ticity. Hence, stochastic systems do, in this interpretation, 
not produce any time at all. Time, again in this interpreta¬ 
tion, is attributed to the nonlinear, non-stochastic action that 
generates information in a complex system. 

Though these remarks are intriguing, the sound physical 
result is contained in the one-subsystem (particle) kinetic 
equation of information which forms the basic equation for 
the evolution of information. We have not given an explicit 
derivation of its right hand side, the correlation functional 
C = K, the Kolmogorov entropy rate. This is left as an ex¬ 
ercise for future research until the hierarchy equations have 
been constructed. Rigorous construction of the hierarchy 
equations is inhibited in this communication by restriction of 
space. We have also neither invested effort into any quantum 
mechanical nor field theoretical formulation which both are 
of utmost interest in applications, nor have we envisaged in¬ 
vestigation of any different more general definition of infor¬ 
mation than the classical Boltzmann-Shannon information. 
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